---
title: Evaluation quickstart
sidebarTitle: Evaluate an application
---

Evaluations are a quantitative way to **measure performance of LLM applications**, which is important because LLMs don't always behave predictably — small changes in prompts, models, or inputs can significantly impact results. Evaluations provide a **structured way to identify failures, compare changes across different versions of your application**, and build more reliable AI applications.

Evaluations are made up of three components:

1. A [dataset](/langsmith/evaluation-concepts#datasets) with test inputs and optionally expected outputs.
2. A [target function](/langsmith/define-target-function) that defines what you're evaluating. For example, this may be one LLM call that includes the new prompt you are testing, a part of your application or your end to end application.
3. [Evaluators](/langsmith/evaluation-concepts#evaluators) that score your target function's outputs.

This quick start guides you through running a simple evaluation to test the correctness of LLM responses with the LangSmith SDK or UI.

<Tip>
This guide uses prebuilt LLM-as-judge evaluators from the open-source [`openevals`](https://github.com/langchain-ai/openevals) package. OpenEvals includes a set of commonly used evaluators and is a great starting point if you're new to evaluations. If you want greater flexibility in how you evaluate your apps, you can also [define completely custom evaluators using your own code](/langsmith/code-evaluator).
</Tip>

## LangSmith SDK

### 1. Install dependencies

<CodeGroup>

```bash Python
pip install -U langsmith openevals openai
```

```bash TypeScript
npm install langsmith openevals openai
```

</CodeGroup>

<Info>
If you are using `yarn` as your package manager, you will also need to manually install `@langchain/core` as a peer dependency of `openevals`. This is not required for LangSmith evals in general - you may define evaluators [using arbitrary custom code](/langsmith/code-evaluator).
</Info>

### 2. Create a LangSmith API key

To create an API key, head to the [Settings page](https://smith.langchain.com/settings). Then click **+ API Key.**

### 3. Set up environment variables

This guide uses OpenAI, but you can adapt it to use any LLM provider.

If you're using Anthropic, use the [Anthropic wrapper](/langsmith/annotate-code#wrap-the-anthropic-client-python-only) to trace your calls. For other providers, use [the traceable wrapper](/langsmith/annotate-code#use-%40traceable-%2F-traceable).

```shell
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your-langchain-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"
```

### 4. Create a dataset

Next, define example input and reference output pairs that you'll use to evaluate your app:

<CodeGroup>

```python Python
from langsmith import Client
client = Client()

# Programmatically create a dataset in LangSmith
# For other dataset creation methods, see:
# https://docs.smith.langchain.com/langsmith/manage-datasets-programmatically
# https://docs.smith.langchain.com/langsmith/manage-datasets-in-application
dataset = client.create_dataset(
    dataset_name="Sample dataset", description="A sample dataset in LangSmith."
)

# Create examples
examples = [
    {
        "inputs": {"question": "Which country is Mount Kilimanjaro located in?"},
        "outputs": {"answer": "Mount Kilimanjaro is located in Tanzania."},
    },
    {
        "inputs": {"question": "What is Earth's lowest point?"},
        "outputs": {"answer": "Earth's lowest point is The Dead Sea."},
    },
]

# Add examples to the dataset
client.create_examples(dataset_id=dataset.id, examples=examples)
```

```typescript TypeScript
import { Client } from "langsmith";

const client = new Client();

// Programmatically create a dataset in LangSmith
// For other dataset creation methods, see:
// https://docs.smith.langchain.com/langsmith/manage-datasets-programmatically
// https://docs.smith.langchain.com/langsmith/manage-datasets-in-application
const dataset = await client.createDataset("Sample dataset", {
  description: "A sample dataset in LangSmith.",
});

// Create inputs and reference outputs
const examples = [
  {
    inputs: { question: "Which country is Mount Kilimanjaro located in?" },
    outputs: { answer: "Mount Kilimanjaro is located in Tanzania." },
    dataset_id: dataset.id,
  },
  {
    inputs: { question: "What is Earth's lowest point?" },
    outputs: { answer: "Earth's lowest point is The Dead Sea." },
    dataset_id: dataset.id,
  },
];

// Add examples to the dataset
await client.createExamples(examples);
```

</CodeGroup>

### 5. Define what you're evaluating

Now, define a [target function](/langsmith/define-target-function) that contains what you're evaluating. In this guide, we'll define a target function that contains a single LLM call to answer a question.

<CodeGroup>

```python Python
from langsmith import wrappers
from openai import OpenAI

# Wrap the OpenAI client for LangSmith tracing
openai_client = wrappers.wrap_openai(OpenAI())

# Define the application logic you want to evaluate inside a target function
# The SDK will automatically send the inputs from the dataset to your target function
def target(inputs: dict) -> dict:
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer the following question accurately"},
            {"role": "user", "content": inputs["question"]},
        ],
    )
    return { "answer": response.choices[0].message.content.strip() }
```

```typescript TypeScript
import { wrapOpenAI } from "langsmith/wrappers";
import OpenAI from "openai";

const openai = wrapOpenAI(new OpenAI());

// Define the application logic you want to evaluate inside a target function
// The SDK will automatically send the inputs from the dataset to your target function
async function target(inputs: { question: string }): Promise<{ answer: string }> {
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      { role: "system", content: "Answer the following question accurately" },
      { role: "user", content: inputs.question },
    ],
  });
  return { answer: response.choices[0].message.content?.trim() || "" };
}
```

</CodeGroup>


### 6. Define evaluator

Import a prebuilt prompt from [`openevals`](https://github.com/langchain-ai/openevals) and create an evaluator. `outputs` are the result of your target function. `reference_outputs` / `referenceOutputs` are from the example pairs you defined in [step 4](#4-create-a-dataset) above.

<Info>
`CORRECTNESS_PROMPT` is just an f-string with variables for `"inputs"`, `"outputs"`, and `"reference_outputs"`. See [here](https://github.com/langchain-ai/openevals#customizing-prompts) for more information on customizing OpenEvals prompts.
</Info>

<CodeGroup>

```python Python
from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

def correctness_evaluator(inputs: dict, outputs: dict, reference_outputs: dict):
    evaluator = create_llm_as_judge(
        prompt=CORRECTNESS_PROMPT,
        model="openai:o3-mini",
        feedback_key="correctness",
    )
    eval_result = evaluator(
        inputs=inputs,
        outputs=outputs,
        reference_outputs=reference_outputs
    )
    return eval_result
```

```typescript TypeScript
import { createLLMAsJudge, CORRECTNESS_PROMPT } from "openevals";

const correctnessEvaluator = async (params: {
  inputs: Record<string, unknown>;
  outputs: Record<string, unknown>;
  referenceOutputs?: Record<string, unknown>;
}) => {
  const evaluator = createLLMAsJudge({
    prompt: CORRECTNESS_PROMPT,
    model: "openai:o3-mini",
    feedbackKey: "correctness",
  });
  const evaluatorResult = await evaluator({
    inputs: params.inputs,
    outputs: params.outputs,
    referenceOutputs: params.referenceOutputs,
  });
  return evaluatorResult;
};
```

</CodeGroup>

### 7. Run and view results

Finally, run the experiment!

<CodeGroup>

```python Python
# After running the evaluation, a link will be provided to view the results in langsmith
experiment_results = client.evaluate(
    target,
    data="Sample dataset",
    evaluators=[
        correctness_evaluator,
        # can add multiple evaluators here
    ],
    experiment_prefix="first-eval-in-langsmith",
    max_concurrency=2,
)
```

```typescript TypeScript
import { evaluate } from "langsmith/evaluation";

// After running the evaluation, a link will be provided to view the results in langsmith
await evaluate(
  target,
  {
    data: "Sample dataset",
    evaluators: [
      correctnessEvaluator,
      // can add multiple evaluators here
    ],
    experimentPrefix: "first-eval-in-langsmith",
    maxConcurrency: 2,
  }
);
```

</CodeGroup>

Click the link printed out by your evaluation run to access the LangSmith Experiments UI, and explore the results of the experiment.

![Gif showing sample dataset, clicking the UI to access the Experiment UI](/langsmith/images/view-experiment.gif)

### Next steps

<Check>
To learn more about running experiments in LangSmith, read the [evaluation conceptual guide](/langsmith/evaluation-concepts).
</Check>

* For more details on evaluations, refer to the [Evaluation documentation](/langsmith/evaluation).
* Check out the [OpenEvals README](https://github.com/langchain-ai/openevals) to see all available prebuilt evaluators and how to customize them.
* Learn [how to define custom evaluators](/langsmith/code-evaluator) that contain arbitrary code.
* For comprehensive descriptions of every class and function see the [Python](https://docs.smith.langchain.com/reference/python/reference) or [Typescript](https://docs.smith.langchain.com/reference/js) SDK references.

Or, if you prefer video tutorials, check out the [Datasets, Evaluators, and Experiments videos](https://academy.langchain.com/pages/intro-to-langsmith-preview) from the Introduction to LangSmith Course.

## LangSmith UI

### 1. Navigate to the playground

LangSmith's [prompt playground](/langsmith/observability-concepts#prompt-playground) makes it possible to run evaluations over different prompts, new models or test different model configurations. Go to LangSmith's **Playground** in the UI.

### 2. Create a prompt

Modify the system prompt to:

```
Answer the following question accurately:
```

### 3. Create a dataset

Click **Set up Evaluation**, then use the **+ New** button in the dropdown to create a new dataset.

Add the following examples to the dataset:

| Inputs                                                   | Reference Outputs                                 |
| -------------------------------------------------------- | ------------------------------------------------- |
| question: Which country is Mount Kilimanjaro located in? | output: Mount Kilimanjaro is located in Tanzania. |
| question: What is Earth's lowest point?                  | output: Earth's lowest point is The Dead Sea.     |

Press **Save** to save your newly created dataset.

### 4. Add an evaluator

Click **+Evaluator**. Select **Correctness** from the pre-built evaluator options. Press **Save**.

### 5. Run your evaluation

Press **Start** on the top right to run your evaluation. Running this evaluation will create an experiment that you can view in full by clicking the experiment name.

![Gif showing sample dataset, clicking the UI to access the Experiment UI](/langsmith/images/evals-quick-start.gif)

### Next steps

<Tip>
To learn more about running experiments in LangSmith, read the [evaluation conceptual guide](/langsmith/evaluation-concepts).
</Tip>

* For more details on evaluations, refer to the [Evaluation documentation](/langsmith/evaluation).

* Learn how to [create and manage datasets in the UI](/langsmith/manage-datasets-in-application#set-up-your-dataset)
* Learn how to [run an evaluation from the prompt playground](/langsmith/run-evaluation-from-prompt-playground)

If you prefer video tutorials, check out the [Playground videos](https://academy.langchain.com/pages/intro-to-langsmith-preview) from the Introduction to LangSmith Course.

## Video guide
<iframe
  className="w-full aspect-video rounded-xl"
  src="https://www.youtube.com/embed/iEgjJyk3aTw?si=C7BPKXPmdE1yAflv"
  title="YouTube video player"
  frameBorder="0"
  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
  allowFullScreen
></iframe>
